Multi-label classification with a reject option

We consider multi-label classification problems in application scenarios where classifier accu-racy is not satisfactory, but manual annotation is too costly. In single-label problems, a wellknown solution consists of using a reject option, i.e., allowing a classifier to withhold unreliabledecisions, leaving them (and only them) to human operators. We argue that this solution can beexploited also in multi-label problems. However, the current theoretical framework for classi-fication with a reject option applies only to single-label problems. We thus develop a specificframework for multi-label ones. In particular, we extend multi-label accuracy measures to takeinto account rejections, and define manual annotation cost as a cost function. We then formalisethe goal of attaining a desired trade-off between classifier accuracy on non-rejected decisions,and the cost of manually handling rejected decisions, as a constrained optimisation problem. Wefinally develop two possible implementations of our framework, tailored to the widely used Faccuracy measure, and to the only cost models proposed so far for multi-label annotation tasks,and experimentally evaluate them on five application domains.